Estimating Continuous Distributions in Bayesian Classifiers

When modeling a probability distribution with a Bayesian network, we are faced with the problem of how to handle continuous vari­ ables. Most previous work has either solved the problem by discretizing, or assumed that the data are generated by a single Gaussian. In this paper we abandon the normality as­ sumption and instead use statistical methods for nonparametric density estimation. For a naive Bayesian classifier, we present experi­ mental results on a variety of natural and ar­ tificial domains, comparing two methods of density estimation: assuming normality and modeling each conditional distribution with a single Gaussian; and using nonparamet­ ric kernel density estimation. We observe large reductions in error on several natural and artificial data sets, which suggests that kernel estimation is a useful tool for learning Bayesian models. 1 Introduction In recent years, methods for inducing probabilistic de­ scriptions from training data have emerged as a major alternative to more established approaches to machine learning, such as decision-tree induction and neural networks. For example, Cooper & Herskovits (1992) describe a greedy algorithm that determines the struc­ ture of a Bayesian inference network from data, while Heckerman, Geiger & Chickering (1994), Provan & Singh (1995), and others report advances on this ba­ sic approach. Bayesian networks provide a promising representation for machine learning for the same rea­ sons they are useful in performance tasks such as di­ agnosis: they deal explicitly with issues of uncertainty and noise, which are central problems in any induction task. However, some of the most impressive results to date have come from a much simpler-and much older-approach to probabilistic induction known as the naive Bayesian classifier. Despite the simplifying assump­ tions that underlie the naive Bayesian classifier, exper­ iments on real-world data have repeatedly shown it to be competitive with much more sophisticated induc­ tion algorithms. For example, Clark & Niblett (1989) report naive Bayes producing accuracies comparable to those for rule-induction methods in medical domains, and Langley, lba & Thompson (1992) found that it outperformed an algorithm for decision-tree induction in four out of five domains. These impressive results have motivated some re­ searchers to explore extensions of naive Bayes that lessen dependence on its assumptions but that retain its inherent simplicity and clear probabilistic seman­ tics. Langley & Sage (1994) describe a variation that mitigates the independence assumption by eliminat­ ing predictive features that are correlated with others. Kononenko (1991) and Pazzani …